Color 3D Model-Based Tracking with Arbitrary Projection Models

We present a colorand shape-based 3D tracking system suited to a large class of vision sensors. The method is, in fact, applicable with any projection model, provided that it is calibrated and the projection function is known. The tracking architecture is based on Particle Filtering methods where each particle represents the 3D state of the object, rather that its state in the image, therefore bypassing the nonlinearity caused by the projection model. This allows the use of realistic 3D motion models and easy integration of the sensor self-motion measurements. All nonlinearities are concentrated in the observation model that, for each particle, projects a few tens of special points onto the image, on (and around) the 3D object’s surface. The likelihood of each state is then evaluated using color histograms. Since only pixel access operations are required, the method does not involve costly image processing routines like edge/feature extraction, color segmentation or 3D reconstruction, that can be cumbersome with omnidirectional projection models. The tracking system copes well with motion and optical blur. We show applications of tracking various objects (balls, boxes) in mobile robots with catadioptric and dioptric omnidirectional sensors.